Using Computer Vision in Real Applications: Two Success Stories,

We present two systems which are used today in industrial applications. While they are very different, and the two domains have no overlap, they share the properties that the problems they address were considered very hard to solve, and the requirements were very constraining, especially complete automation. We jirst present a system which automatically registers two sets of halfone color separations used to produce color pictures. The challenges involve accuracy, speed of processing, and consistency with human operators. The second system substitutes, in real-time, a given billboard in a video stream, with another, synthetically generated billboard. The challenges involve real-time performance and photo-realism. For each of the two systems, we provide some background, describe the requirements and the issues, then the implemented solution. 1 Registration of Color Separations 1.1 Background In order to print color photographs such as those which appear in mass circulation magazines, or in other publications where quality is important, the original color image is broken down into four (or more) separate photographic images, which are then processed independently. Each one is a halftone picture, in which the appearance of continuous tones is obtained by using dots of varying sizes. Each color separation cames black and white tone information which reflects the corresponding intensity contents of the original color picture. It is crucial for the halftone color separations to be very accurately (within 25p or better) registered, so that the printed image faithfully reproduces the original. Currently, this registration is for the most part performed manually by highly trained personnel, referred to in the printing industry as "strippers". The manual process involves taping the separation films to a larger, clear polyester film called a "camer sheet". The first (reference) film is taped to a pre-punched camer sheet; the following films are visually aligned to the reference and taped onto their own pre-punched carrier sheet in their registered position.This manual registration suffers from some shortcomings: accuracy: strippers need to consistently register separation films within a fraction of the interdot spacing, and this is difficult to maintain, even for experienced strippers. In high quality printing, there are at least 5.9 dots per mm (150 dots per inch), so the center to center distance is less than 0.19rnm (7 mils). With such tight constraints, human errors inevitably occur. Furthermore, if an unacceptable registration is not discovered until press time, the presses have to be held up, an inordinately expensive event. cost: manual registration is a tedious, slow, and highly labor intensive process. It typically takes a highly skilled (and therefore well paid) stripper between ten and twenty minutes to register four color separations. Consistency and reliability are hard to maintain as skill and vision vary from one stripper to the next, and each stripper is subject to fatigue. magnification aids have been proposed to increase accuracy. The magnified image of a small portion of a negative often appears as a rather random collection of dots, which do not correlate from one separation to the next. This problem is aggravated by the difficulty in accomplishing fine eye and hand controlled motion when viewing a magnified image. Magnification aids are therefore mostly used for the verification of registration. 1.2 Requirements and Challenges The previous discussion sets the stage for the requirements of a successful vision-based system to perform the color registration automatically: Speed: the system should be at least as fast as a human stripper and therefore process about 10 sets per hour, including all human intervention (rough register, taping to carrier sheet, loading and unloading sheets and verification) and all processing steps (image acquisition and processing, matching, mechanical motion and hole punching). Accuracy: the machine should consistently produce registration results of less than 25p (1 mil) for halftone details and of less than 12p (0.5 mils) for reverse type (letters or register marks). Reliability: the overall system, including mechanical parts, must be reliable because of the very tight deadlines imposed by magazines. (Overnight runs are more the norm than the exception.) Also, it is essential for the results to be consistent with the ones an operator would have manually selected. This means that there should be very few instances in which the machine aborts (refuses to punch because of inconsistencies) or punches films that would later be considered unacceptable for printing. Finally, there exist cost considerations which impact the design in terms of reasonable computer resources. and acceptability issues in a well established industry. A solution to these problems is described in the text of two recent patents[ 1 ] [2]. In machine vision terms, the problem is to match two images, typically 20 x 25 cm2 (8"x10), with 1/1000 in accuracy or better. These images, however, are halftone, which means that they are already coded on a grid, with each dot size proportional to the corresponding intensity. From one color separation to the next, the dots are not overlapping because the grids are rotated between colors (to avoid moirk effects). In fact, corresponding dots in registration form a rosette pattern. As a result of this situation, methods based on the correlation of intensity values are limited: if the resolution is coarse enough, pixels will indeed correspond to each other, but the accuracy will not be sufficient; if the resolution is finer, then pixels do not correspond to each other. These limitations are compounded by the fact that the information encoded by different colors, although globally similar, may be locally very different. An area can be dark in one color separation and light in another; text (which is easier to match) generally does not occur in all colors. It also may happen for the same pattern in different color separations to appear with different sizes (choke and spread effect). 1.3 Implemented Solution Three issues are involved in the process described above: representation, matching and verification. Representation refers to the selection of features and to the procedures used to extract these features from the input images. Matching refers to the procedures applied to each pair of images to obtain a measure of their relative mis-registration. Verijication refers to procedures used to verify the relative mis-registration, to estimate rotation, and to the subsequent decision to punch registration holes to the films. We illustrate the problems on a specific example. Figure l(a) and (b) show the cyan and magenta images corresponding to the 7"xll" negatives. Following the strategy used by human strippers, an operator roughly aligns the films to the reference, and selects two detail areas on the reference, with a cursor. The machine then takes over, automatically extracts features in the windows, matches them, and verifies the quality to either punch registration holes or reject the set of films. This is illustrated in Figure 2. Feature Extraction As noted earlier, windows in corresponding areas of two separations can appear very different, as shown in Figure 3. We therefore use as matching primitives the macro edges between regions of different dot density. They are extracted as subpixel zero-crossings of the convolution of the image with a large Laplacian of Gaussian mask[4], where the space constant is a function of the screen resolution. This computation is performed efficiently[3]. These contour points are then linked, thresholded using hysteresis, and approximated by linear segments. Window Matching film. If these two vectors differ in length by more than By establishing correspondences between features 3 mils, the punching procedure is aborted. extracted from a window, we obtain an estimate of translation (or mis-registration) in the two areas of the The machine film, and then combine these results to estimate the The machine performs the registration of a four color rotation in the whole image. set of films in about 5 minutes, including all operations. This corresponds to a cycle time of 12 seconds The translation between two windows needs to be to match two windows. This speed is achieved by estimated robustly, as only subsets of features pair up. We use a Hough-like voting scheme in translation using a Mercury array processor to perform the vector space, adapted for linear segments. operations. The code is written in Fortran, with some critical inner loops microcoded for efficiency. Good matches produce a sharp peak, whose position gives the estimate of translation. We post process the results to handle choke and spread effects, which produce crater-like peaks.